Analysis and Optimization of the Hadoop Speculative Execution Mechanism

The existing Hadoop clusters are mostly composed of heterogeneous nodes, which have different computing and storage capacities, with the speed of maps to reduce tasks performed on the nodes being quite different. However, the finish time of the entire job is determined by the slowest task, so looking for the “drag tasks” strategy has a dominant position in the whole job scheduling process. The current speculative execution mechanism of Hadoop results in shortcomings to find the drag tasks in time. In this paper we try to improve the speculative execution mechanism, and then we apply the improved First-in-First-out (FIFO) scheduler to the Hadoop cluster. Experiments verify that the improved mechanism has better performance when the Hadoop platform has many drag tasks, where it can increase the cluster resource utilization and throughput.